Django + SQLite + Collations

There is rather well-known "SQLite case-insensitive problem", which is that SQLite can do case-insensitive SELECT only for ASCII character set. This problem was discussed at Django users list, posted as Djungo feature request and documented in the documentation:

  SQLite doesn't support case-insensitive matching for non-ASCII strings. Some 
  possible workarounds for this are `documented at sqlite.org`_, but they are 
  not utilised by the default SQLite backend in Django. Therefore, if you are 
  using the ``iexact`` lookup type in your queryset filters, be aware that it 
  will not work as expected for non-ASCII strings.

I also met this problem and I had to find a solution since I need to deploy one small application (ebooks catalogue) at the department machine where I didn't have root account and was unable to install another database engine.

The solution is not really hard.

Support of collations for SQLite is provided by loadable extension through ICU library, which is installed on almost every Linux box. This is documented here.

So first step is to get ICU extension for SQLite here and compile it with

  gcc -shared icu.c icu-config --ldflags -o icu.so

You need to have development files of libicu installed, for Debian/Ubuntu do

  sudo apt-get install libicu-dev

Now it is possible to test the extension (and also check if sqlite3 in your system is built with extension support):

  nazarov@heps1:~/lib> sqlite3
  SQLite version 3.6.11
  Enter ".help" for instructions
  Enter SQL statements terminated with a ";"
  sqlite> .load ./icu.so
  sqlite>  SELECT icu_load_collation('ru_RU', 'RUSSIAN');
  
  sqlite> SELECT * FROM core_author WHERE name LIKE 'вайн%';
  1420|Вайнер А.Л.
  4557|Вайнберг М.М.
  4558|Вайнберг М.М.
  6374|Вайнберг Б.
  7326|Вайнштейн Б.К.
  7327|Вайнштейн Л.А.
  7458|Вайнштейн С.И.
  8636|Вайнберг С.
  8958|Вайнштейн С.И.

If your SQLite is built without extension support, get appropriate version at SQLite homepage.

Then we need extension support in pysqlite. Since version 2.5.2 pysqlite can load SQLite extensions, but this feature is disabled by default. So download pysqlite, extract, remove the line

  define=SQLITE_OMIT_LOAD_EXTENSION

from the file setup.cfg and build it:

  wget http://oss.itsystementwicklung.de/download/pysqlite/2.5/2.5.5/pysqlite-2.5.5.tar.gz
  tar xzf pysqlite-2.5.5.tar.gz
  cd pysqlite-2.5.5
  vi setup.cfg
  python setup.py build

Now you can check, that pysqlite can load icu and make case insensitive selects:

  export PYTHONPATH=`pwd`:$PYTHONPATH;export LD_LIBRARY_PATH=`pwd`:$LD_LIBRARY_PATH;python
  Python 2.5 (r25:51908, Nov 27 2006, 19:14:46) 
  [GCC 4.1.2 20061115 (prerelease) (SUSE Linux)] on linux2
  Type "help", "copyright", "credits" or "license" for more information.
  >>> from pysqlite2 import dbapi2 as sqlite3
  >>> con = sqlite3.connect("lib.db")
  >>> con.enable_load_extension(True)
  >>> con.load_extension("./icu.so")
  >>> con.execute("SELECT icu_load_collation('ru_RU', 'RUSSIAN');")
  <pysqlite2.dbapi2.Cursor object at 0xb7cab660>
  >>> con.execute("SELECT * FROM core_author WHERE name LIKE 'вайн%';")
  <pysqlite2.dbapi2.Cursor object at 0xb7cab820>

Also I had to build static version of pysqlite which included SQLite, because SQLite on target system didn't support loadable extenstion. I did it with

  python setup.py build_static

So now we need to tell Django to use collations. Correct way would be to write custom db backend, but it is far simplier just to add 3 following lines to the file django/db/backends/sqlite3/base.py:

*** 143,148 ****
--- 143,153 ----
              }
              kwargs.update(self.options)
              self.connection = Database.connect(**kwargs)
+             # Add Russian collation
+             self.connection.enable_load_extension(True)
+             self.connection.load_extension("/home/nazarov/lib/icu.so")
+             self.connection.execute("SELECT icu_load_collation('ru_RU', 'RUSSIAN');")
+ 
              # Register extract, date_trunc, and regexp functions.
              self.connection.create_function("django_extract", 2, _sqlite_extract)
              self.connection.create_function("django_date_trunc", 2, _sqlite_date_trunc)

Now Django applications can do case-insensitive SELECTs (iexact, icontains etc) for Russian language.

On my target machine I had to start Django development server with

  export PYTHONPATH=`pwd`/Django-1.0.2-final/:`pwd`:$PYTHONPATH
  export LD_LIBRARY_PATH=`pwd`:$LD_LIBRARY_PATH;
  python library/manage.py runserver 10.0.0.15:8000

since I could not do system-wide install.

Popular history of particle accelerators experiments (part 1)

Large Hadron Collider (LHC) will start operation this year, so there are a lot of fuss in press and blogs. Some people try to obtain public attention with legal claims against CERN. Nonetheless, general public is not aware of particle physics and accelerator experiments, and conceive LHC as final ultimate device for discovery of theory of everything or even as a doomsday machine. I want to review the history of particle physics experiments to show that LHC is next step in research. It shouldn’t be viewed as final or outstanding.

My posts are based on lectures of I.N. Meshkov at sixth winter school "Introduction to theory of fundamental interactions" in Dubna.

Ancient Greeks proposed two opposite ideas of structure of matter. One conception was that everything consists of several kinds of indivisible particles while the other told that everything could be divided into infinitely small parts.

So the target of particle physics is to find out which point of view is closer to real world and to study properties of particles for better understanding of our world which consists of particles.

Today we know that there are several kinds of elementary particles, which are united into so called “Standard model”.

---

Рис. 1: Standard model of elementary particles

---

String theory tell us that all these particles are modes of oscillation of more elementary objects, which are called "strings but now it is just a hypothesis which could not be checked with experiment.

History of particle physics started at 1895 from discovery of photons as X-rays particles by Konrad Rentgen and discovery of electron by Joseph John Thomson (1897). First experiments didn’t use particle accelerators.

---

Рис. 2: Konrad Rentgen Рис. 3: Joseph John Thomson

---

First electrostatic accelerators were created in 1931-1932 by R. Van-der-Graaf and J.D. Cockroft with E.T.S. Walton, and allowed the discovery of neutron by James Chadwick. He used reaction

These early accelerators were not very powerful and sophisticated, so many discoveries in particle physics were made from observations of cosmic rays, which started at 1912 by, Victor Hess. He has found in experiment with 3 electrometers carried out in a free balloon flight up to an altitude of 5300 meters that the ionization rate increased approximately fourfold over the rate at ground level. He concluded :"The results of my observation are best explained by the assumption that a radiation of very great penetrating power enters our atmosphere from above".

In 1932, Carl D. Anderson discovered positron by passing cosmic rays through a "Wilson chamber"with lead plate surrounded by a magnet to distinguish the particles charge. It was the beginning of the competition of accelerators with cosmic rays.

Cosmic rays consist of particles produced in outer space in thermonuclear reactions in stars and during supernova bursts. Some of these particles have energy which greatly exceeds energy of LHC and all the accelerators possible in near future. So it is one of the reasons why LHC wouldn’t destroy the Earth. But in spite of such enormous energy cosmic rays are rather inconvenient for particle physics experiments, because the density of such high-energy particles is very low.

In 1929-1932 Ernest O. Lawrence invented the resonance acceleration method which allowed construction of first cyclotron and first resonance linear accelerator, but next advance in particle physics was achieved in cosmic rays observations at 1936. Carl D. Anderson discovered of "mesotron which was supposed to be hypothetical Yukawa particle carrying interaction between protons and neutrons in nuclei. Later it was found out that mesotron was not Yukawa particle, but other electron-alike fundamental particle, so it was renamed to muon.

Break-through in accelerator technics was made by Vladimir I. Veksler (USSR, 1944) and Edwin McMillan (USA, 1945) with the discovery of the “Phase stability principle” and construction of the first electron synchrotron (Veksler) and the first synchrocyclotron (E.McMillan).

---

Рис. 4: Vladimir I. Veksler Рис. 5: Edwin McMillan Рис. 6: Vladimir I. Veksler and Edwin McMillan at JINR (Dubna) 1958

---

Cosmic rays observations also made big progress, in 1947 pions, -hyperon and kaons were discovered. But cosmic rays observations were not very accurate, so there were some false “discoveries”, such as “Varitrons”. So first synchrocyclotrons allowed to improve results of cosmic rays observation.

In 1947 Berkeley Synchrocyclotron was put into operation. It had 184 inches (4.7 m) in diameter, and energy of accelerated -particles of 400 MeV. On this accelerator Lattes and E. Gardner achieved first artificial production of pions in reaction

Similar synchrocyclotron “Phasotron” was built in Dubna (USSR) at 1949. It is still in operation now and is used for cancer therapy, muon-catalysis, pion and muon physics experiments and nuclear physics experiments.

---

Рис. 7: Phasotron, JINR (Dubna)

---

Since the end of 1940-th accelerators are the main tools of particle physics and majority of experimental results of fundamental character has been obtained both in particle, and nuclear physics at accelerators.

First day in our office

Our project is getting real.
At Saturday we had the first day in our new office at the outskirts of Saint-Petersburg.
We have big room in the building of former headquarters of an aluminium plant.


Our first tasks were to set up LAN and internet connection, assemble office chairs and put up a whiteboard.



You can see that we successfully fulfilled these tasks!

Business-consumer social networks presentation

Presentation byYury Lifshits on the project of creation of prototype of social network for interaction between businesses abd consumers.

| View | Upload your own

Project page

Half a year of absence

I haven't written anything here for half a year.
It's a pity, but I had not much of enthusiasm for writing, and all of it I put into writing my master's thesis. So diploma is almost finished, and now I am preparing presentation.
Also I've presented my results at the local conference.
You can view my thesis and presentations here (sorry, in Russian only).
The main result is that we can conclude that QFT on non-commutative spaces is sufficiently non-trivial even in simplest case, studied (with opposite results) by Fiore and Wess.
Also I was able to construct several new non-commutative spaces.
Somehow I raised the entropy, because now we can study not one but several examples of non-commutative spaces, constructed with twisting :)

First bits of PyQt

Yesterday I wrote my first PyQt application. It is really simple program, called "tray", which lives in system tray and notifies user of new messages at my favorite social network Vkontakte.ru (Russian clone of FaceBook.com).
The program is less than 170 lines of code. You can try it if you are registered at vkontakte.ru, but the program doesn't matter now.
I am very pleased with simplicity of creation of portable rich GUI applications with PyQt, but as always there are some issues on Windows. I created the program on my favorite old Linux box with vim, so there I had no problems with PyQt4 installation and no need of packaging, because the program consists of one file and three icons.
Then I decided to try it on Windows. First of all I installed newest PyQt 4.3.0. This new version is now packed in self-installing archive, which includes Qt and Eric4 IDE, so now you shouldn't have problems which I described in this post.
After that the program worked perfectly, but I needed to create standalone version for people who don't have Python and PyQt on their PCs.
This is rather simple task, but I've spent several hours fighting some issues.
I used py2exe to create standalone executable. The problem was to include PyQt into it. The idea is simple: I needed to specify that PyQt should be included. So I used solution from py2exe.org. I created file called setup.py with following contents:

from distutils.core import setup
import py2exe
setup(windows=[{"script":"tray.py"}], options={"py2exe":{"includes":["sip"]}})

Then I ran python setup.py install py2exe.
But I received errors like "module import failed: _qt". So I decided to mention inclusion explicitly: python setup.py install py2exe --includes=PyQt4,sip. It didn't help, but error message changed to "cannot find QApplication". Then I tried to reorganize imports in my code:

import sys
from PyQt4.Qt import *
from PyQt4 import QtCore, QtGui

gave correct result, and the executable was created.
I believe that this problem with imports is temporary and will be fixed in near future.

So then I used Inno Setup to create setup package and it was really easy.

Conclusion
It is really easy to create portable applications with great GUI with PyQt4.
The only trade off is the size of installed application on Windows: for my 170 lines of code project I've got 5 Mb setup package, which extracts into 17 Mb application.
The reason is simple - lack of common packaging system with dependencies on Windows, so the program needs it's own copy of Python and PyQt.

My PCs

Today I will try to retrospect history of my home PCs. It is rather personal, so please leave the page immediately if you are not interested :)

For those, who haven't left yet.
My father is a software engineer, so there is nothing unusual that we'd got first home computer as soon as was possible. It was in 1990 or 1991, when the Soviet Union still existed, so there were some problems with foreign devices. Our first computer was hand-made ZX Spectrum clone, which was assembled from different foreign and Russian chips by my father's friend. It had rather peculiar hand-made keyboard, and used Kometa-212 tape-recorder for data storage:
,
and monochrome TV Kaskad-203 as the display, which looked like this:

I'd just started to play chess in local center for youth creativity and enjoyed playing with this computer very much. I was unable to beat it at that time, since I was only six or seven years old :)
Also I tried to do some programming in built-in Basic, but without much success, since I was unable to save entered program on the tape.
I will try to find and photograph remains of that computer.

It's a pity that couple of years after our old TV ceased to function, so we bought another TV, which was incompatible with computer output.

So for couple of years we had no working computer at home. I should mention, that the beginning of 1990s was rather hard time in Russia, so IBM PCs were to expensive for us. They cost more than $1000, and it was very big money. The average wage at that time was about 50-100 USD a month.

When I was eleven, I started study programming in Palace of Youth Creation. I was taught Turbo Pascal 7.0. So I was really happy when we bought a computer as a New Year gift at the end of 1996.

It was rather good and expensive machine with AMD K5 75 MHz processor, 8 megabytes of RAM, 600 megabytes hard drive and brand new Windows 95 OS and 14" CRT monitor GoldStar 1468.

I still use this monitor, which has rather decent picture quality and 800x600/75Hz resolution.

We also bought a printer HP DeskJet 400, which is really cool, I used it last time just two years ago, when I had no time to buy a cartridge for new printer.

I enjoyed playing Warcraft II and programming in Turbo Pascal. But the OS was a complete disaster, it showed Blue Screen of Death several times a day. May be it was due to the lack of system resources, since OS became more stable when we added 8 more megabytes of RAM a year later, or may be because the system was cracked, not bought legally. As you remember, couple hundreds of bucks were a lot of money in Russia in that time :)

In the end of 1997 we added 8 megs of RAM, sound card, CD-ROM and 1.2 Gb hard drive to our PC. Nevertheless, by the end of 1998 it became completely obsolete.
So we "upgraded" it by replacement of main board, processor, RAM, and hard drive. We used the box of old PC, and also bought new LG Flatron 795FT monitor.

New computer was pretty cool, it was based on AMD K6-2/350Mhz processor, with 64 megs of RAM (128 more megabytes was added a year later), 6 Gb Seagate Medalist hard drive and great USR Courier 56k modem. Modem was really good on our old phone line. We had almost no breaks, but rather slow (33,6k) connection because of old hardware at local exchange.

In 1999 computer section in Palace of Youth Creation switched from DOS/Windows 3.11 to FreeBSD, so I started to study C, C++ and Perl, and installed RedHat Linux 5.2 and later 6.0 and 6.2 on our first hard drive, which was 600 megabytes. It's funny that there were enough space for X11, KDE, XEmacs, LaTeX, kernel sources and so on. I also enjoyed Fallout 2 game.

In the end of the year 2000 we decided that we need new PC, so we bought completely new AMD Athlon XP 1733+ MHz with 256 megs of RAM and 40 Gigs IBM hard drive. We plugged newer LG monitor into new PC, and old GoldStar to older one. So for the first time we had more than one working computer :)

Well, later we added new hard drives to new PC, 512 megabytes of RAM, CD-RW/DVD-RW drives, new Canon PIXMA iP4000 printer, Epson scanner and so on, but that is not very interesting. In the beginning of this year 40 Gb IBM hard drive on older PC failed, so that computer became Linux only box. I installed KUbuntu 7.04 on it and I use it now everyday for software development, and it is rather fast :) It can host Apache, MySQL and PHP, also I do C++ and Python development on it. It even can play Youtube movies :) The only thing that I've done for optimization was the replacement of KDE with Fluxbox.